1. Field of the Invention
This invention relates to shared memory based symmetric multiprocessor systems, and more specifically, to an apparatus and method for partitioning and managing memory in a shared memory based multiprocessor system into independent, fault contained domains.
2. Description of Prior Art
Modern computer systems are increasingly comprised of symmetric shared memory based multiprocessor systems (SMPs). SMPs are regularly partitioned and physical resources, such as processors and memory, are assigned to partitions each executing their own operating system. For reliability, availability and serviceability reasons, memory assigned to a partition must be protected from being accessed by other partitions.
The translation and protection mechanisms provided by individual operating systems can be either accidentally or maliciously circumvented to allow access to memory assigned to other partitions. Conceptually this problem can be solved by intercepting all bus traffic and subject this traffic to access verification through determining whether a particular processor is allowed to access a particular memory segment. However, modern cache coherent symmetric multiprocessor systems such as the Intel Pentium Pro and Pentium II enforce very tight timing constraints on their bus to alleviate the problem of bus occupancy.
In particular, cache snooping protocols do not allow any extra cycles to intercept the address put on the bus, verifying it and aborting a transaction upon any access violation, all before the snooping of other processors starts. Typically, once the address is visible on the address bus, the next signal latch initiates the cache snooping. If indeed a processor puts an address outside its assigned memory onto the bus, this request could be filled by a processor outside the originating partition, resulting in invalid cache states, e.g., inter-cache transfers with cache invalidates, leading to inconsistent memory state that cannot be recovered from.
While mechanisms exist that rely on changes to the processor core and the bus architecture, these are typically limiting in terms of addressability and in terms of establishing cache coherent shared memory between the partitions for the purpose of cache coherent inter-partition communication. Having thus given a general overview of the problem area, what is hence needed is an apparatus and a method that provides fault contained memory partitioning while preserving cache coherence domains.
Some attempts at addressing problems similar to these being solved by the present invention are introduced below.
1. A commonly owned, co-pending U.S. patent application Ser. No. 09/256,035 entitled xe2x80x9cSecure Partitioning of Shared Memory Based Multiprocessor Systemxe2x80x9d filed on Feb. 23, 1999, describes an apparatus which establishes cache coherence domains in an SMP node. That apparatus replicates the internal system bus and uses a configurable crossbar switch to connect each of the system components, such as processors, I/O controllers and interrupt controllers to one of the internal busses. All components connected to the same internal bus form a coherence domain.
The apparatus further utilizes memory controller modifications to re-map the real addresses on each internal bus to physical memory. This system was designed to provide 0-based memory to each partition in order to avoid system software changes. It may establish non coherent shared memory regions between partitions by relocating certain real address ranges of different partitions into the same physical memory.
That invention specifically circumvents the problem of domain protection on the same bus, and is further limited by the pin-count of the crossbar switch.
2. U.S. Pat. No. 5,796,605 issued Jul. 2, 1996 describes a technique for system memory space address mapping in a multiprocessor computer system. The disclosed mapping architecture may be applied to a multiprocessor computer system having SMP nodes, where each processing node may include multiple processors. The system memory address space is split into different regions such that each of n SMP nodes is assigned 1/n of the total address space. By assigning 1/n of the global shared memory region to each node, it establishes memory locality that is used in a specific cache coherency protocol to utilize this locality based on the state of operation. In this regard this reference does not deal with partitioning for the purpose of establishing different fault protected system partitions, it neither provides memory partitioning on a single SMP node nor provides inter-partition shared memory regions.
3. U.S. Pat. No. 5,845,071 issued Dec. 1, 1998 describes the partitioning of a multi-node multiprocessor system with globally shared memory into groups of nodes called error containment clusters of nodes or ECCNS. The nodes would be partitioned such that an ECCN resides on a column of nodes or a row of nodes. Within each ECCN there is coherent memory sharing. Between the ECCNs, the communication is through a messaging protocol. The memory within each node is also partitioned into protected and unprotected memory. Unprotected memory is used for messaging and protected memory is used for sharing. A failure in an error containment cluster would corrupt the memory within that cluster, specifically the protected memory within that cluster and also the unprotected memory used by that cluster to communicate with the other clusters. However, the other clusters could continue to run because their protected memory would be unaffected, and could continue to communicate through the remaining unprotected memory.
This patent deals with partitioning clusters and not with partitioning of a single SMP node. It establishes protection domains along SMP boundaries and as such does not deal with partitioning a single SMP. Furthermore, it sets aside special unprotected memory coupled with message passing for inter partition communication to avoid the loss of coherency state that arises when one of the nodes become inoperable.
4. U.S. Pat. No. 3,827,029 dated Jul. 30, 1974 describes a hardware memory violation protect subsystem that may be added to a computer system as a hardware option. The memory protect subsystem includes hardware which may operate in parallel with the computer system memory subsystem and which monitors each attempt to alter data within the memory subsystem. Any attempt to alter data within a protected region may be defeated. Following such an attempt, program execution is interrupted and program control is transferred to the computer system executive software. Although this patent addresses memory protection, it does not address issues of partitioning or problems arising due to the presence of caches and cache coherency traffic, namely illegal inter-cache line transfers.
5. U.S. Pat. No. 4,843,541 issued Jun. 27, 1989, describes a method of logically partitioning an IBM S/370XA Mainframe computer, and requires support from hardware, software and I/O devices. Absolute and virtual addresses of the different operating systems, as well as page addresses for any expanded storage, are relocated into, their assigned partitions. However, this patent does not establish multiple SMP protected memory domains in the same coherence domain on the same bus.
6. U.S. Pat. No. 4,814,982 dated Oct. 9, 1987, describes a system that assigns identifiers to individual processors or input/output (I/O) module for use in controlling access to global memory. The primary focus of this patent is to achieve fault tolerance by replicating processing function and detecting faults to allow takeover on another processor. This patent uses a modified memory controller that separates memory ranges via a processor/task identifier.
However, this patent does not deal with cache coherency issues resulting from executing multiple system images on the same shared bus; the protection is built into the devices attached to the bus, thus at a cost of providing such mechanisms in the devices, faulty addresses do not show up on the bus. Inter-partition communication is not dealt with by this patent. Finally, a number of segments are required to equal the number of processing elements.
The present invention introduces an apparatus and a method for providing fault contained memory partitioning in a cache coherent, symmetric shared memory multiprocessor system while enabling fault contained cache coherence domains as well as cache coherent inter partition memory regions. The entire system may be executed as a single coherence domain regardless of partitioning, and the general memory access and cache coherency traffic are distinguished.
All memory access is intercepted and processed by the memory controller. Before data is read from or written to memory, the address is verified and the executed operation is aborted if the address is outside the memory regions assigned to the processor in use. Accordingly, when this happens, the offending processor is stopped. The inventive apparatus and method utilizes a per partition memory access map, identifying the ranges of memory that a particular processor may access. By allowing overlaps in these access maps, cache coherent inter partition shared memory regions may be established, to facilitate the efficient implementation of shared locks and other inter-partition communication protocols.
Consistency of the cache coherency domains is achieved through a protocol performing address verification concurrently with the bus snooping protocol. Rather than verifying the address when a transaction is put on the bus and then signaling validity of the address on the bus to other processors, the apparatus and method of the present invention snoops on requests on the bus without interfering with them. Due to the tight timing constraints on the bus, nothing can be done to these transactions.
Concurrently with the request being snooped on by all processors, the verification device determines whether the access is valid. By the time a response is issued by one of the processors, the address verification has determined whether the address is valid. A copy of data of the response is kept. In the case of an invalid access the issuing processor is immediately stopped and the memory hierarchy of the responding processor is repaired with a copy of data. This protocol has the advantage that the tight timing constraints of the communications bus snooping protocols are not violated. The advantage of this invention is that the core of symmetric multiprocessor systems, namely the processors and their built in caches, as well as the communications bus tailored to these processors, may be reused since all modifications are contained with in the memory controller.
In contrast to commonly owned, co-pending U.S. patent application Ser. No. 09/256,035 entitled xe2x80x9cSecure Partitioning of Shared Memory Based Multiprocessor Systemxe2x80x9d filed on Feb. 23, 1999, the whole contents disclosure of which is incorporated herein by reference, the present invention runs the entire SMP as a single cache coherence domain with no modifications to the system communications bus and no replication of the system communications bus. It is able to establish and provide cache coherent shared memory regions for inter partition communication. However, it does not provide 0-based memory to each partition and thus requires that the operating system software is relocatable.
The invention may use address verification mechanisms similar to the real-to-physical reaping device of Ser. No. 09/256,035, as address ranges on the communications bus must be recognized in both inventions. To that extent, similar mechanisms may be used to determine an address range hit. However, the outcome of an address range hit is utilized differently; where Ser. No. 09/256,035 uses an address range hit to relocate the real address into a physical address for memory access, the present invention uses the lack of an address range hit to abort the transaction and to reset the offending processor.